Problem

Our current package-release CI is broken. It requires manual wheel-publish as at
https://github.com/single-cell-data/TileDB-SOMA/wiki/PyPI-packaging-notes

CI failures are like
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9189052757
following the well-intentioned but fateful #2491.

Also nightly wheel-build failures such as #2630.

Cause

Debugging reveals:

• [ci] Move Python wheel-builds from macos-11,macos-12 to macos-12,macos-latest #2491 moved Python wheel-builds from macos 11 and 12 to 14 and 12
  • Why we did that: GitHub Actions is deprecating macos 11 runners
• Printing python -c 'from distutils import util; print(util.get_platform())' at build-wheel time and smoke-test-wheel time
  • Build-wheel time on macos 14: this reports (!) macosx-10.9-universal2
    • https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193078472/job/25283290847#step:4:20
  • Smoke-test-wheel time on macos 14: this reports (!) macosx-10.9-universal2
  • https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193078472/job/25284241544#step:4:93
• However: the wheel that's created has the substring macosx_11 in its filename :(
  • https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193078472/job/25283290847#step:5:1005
  • Created wheel for tiledbsoma: filename=tiledbsoma-1.11.2.post0.dev4180921687-cp38-cp38-macosx_11_0_x86_64.whl
• As a result, at smoke-test time, util.get_platform() says macosx-10.9-universal2 and yet there is no wheel for 10.9 -- only 11.0
  • On MacOS 11, pip install of a wheel with either macosx_10_9 or macosx_11_0 in its wheel-file name succeeds
  • Not so on MacOS 14
  • So this all worked fine when the smoke-test runner was macos 11; not fine now that it's macos 12 or 14

Attempted solutions

• ❌ Setting the CMAKE_OSX_DEPLOYMENT_TARGET: "10.9" in our CI YAML's env does not help since in [c++] Refactor SOMAArray and SOMAGroup to inherit from SOMAObject #2124 we set -mmacosx-version-min=11.0 to support usage of C++17 feature std::filesystem::path, which overrides the env
• ❌ Having our CI YAML copy files like tiledbsoma-VERSIONHERE-cp311-cp311-macosx_11_0_arm64.whl to tiledbsoma-VERSIONHERE-cp311-cp311-macosx_14_0_arm64.whl does not work, and in fact is a step in the wrong direction: since macos14 self-reports as 10.9 within pip install as shown above.
• ❌ Having our CI YAML rename files like tiledbsoma-VERSIONHERE-cp311-cp311-macosx_11_0_arm64.whl to tiledbsoma-VERSIONHERE-cp311-cp311-macosx_10_9_arm64.whl does work, since macos 14 self-reports as 10.9 at pip install time.
  • ✅ Works on my laptop with macos14 and Python 3.9
  • ❌ Fails in CI with macos14 and Python 3.11
  • ❌ Fails in macos-on-macos Docker on my laptop with Python 3.11 (see below)

Laptop details

With macos 14 and Python 3.9:

laptop$ sw_vers
ProductName:		macOS
ProductVersion:		14.5
BuildVersion:		23F79

laptop$ python --version
Python 3.9.13

laptop$ ll | grep macos | grep x86
-rw-r--r--@ 1 johnkerl  staff  24547997 May 21 22:50 tiledbsoma-1.11.2-cp310-cp310-macosx_11_0_x86_64.whl
-rw-r--r--@ 1 johnkerl  staff  24549656 May 21 22:50 tiledbsoma-1.11.2-cp311-cp311-macosx_11_0_x86_64.whl
-rw-r--r--@ 1 johnkerl  staff  24547829 May 21 22:50 tiledbsoma-1.11.2-cp38-cp38-macosx_11_0_x86_64.whl
-rw-r--r--@ 1 johnkerl  staff  24548029 May 21 22:50 tiledbsoma-1.11.2-cp39-cp39-macosx_11_0_x86_64.whl
-rw-r--r--@ 1 johnkerl  staff  97615624 May 21 18:53 wheels-macos-x86_64.zip

laptop$ pip install tiledbsoma-1.11.2-cp39-cp39-macosx_11_0_x86_64.whl
ERROR: tiledbsoma-1.11.2-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform.

laptop$ cp tiledbsoma-1.11.2-cp39-cp39-macosx_11_0_x86_64.whl tiledbsoma-1.11.2-cp39-cp39-macosx_10_9_x86_64.whl

laptop$ pip install tiledbsoma-1.11.2-cp39-cp39-macosx_10_9_x86_64.whl
Processing ./tiledbsoma-1.11.2-cp39-cp39-macosx_10_9_x86_64.whl

Note this exact logic -- successful on laptop -- fails as of
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9198109199/job/25300662543
for commit d793416

Using https://hub.docker.com/r/sickcodes/docker-osx -- macos-on-macos Docker -- with Python 3.11 I found that the trick which worked on my laptop (macos 14 with Python 3.9), and which fails in CI here with macos 14 and Python 3.11, also fails in sickcodes/docker-osx with Python 3.11.

In short it seems that the renaming trick from https://stackoverflow.com/questions/65888506/error-wheel-whl-is-not-a-supported-wheel-on-this-platform works in Python 3.9 but not 3.11.

Also note: supposedly-macos-on-macos sickcodes/docker-osx:latest isn't really:

[arch@041a7b1338f2 OSX-KVM]$ uname -a
Linux 041a7b1338f2 5.10.104-linuxkit #1 SMP Wed Mar 9 19:05:23 UTC 2022 x86_64 GNU/Linux

Also, why doesn't pip install tiledbsoma (from PyPI) fail on my Mac?

johnkerl@exade[prod][][~]$ pip install tiledbsoma
Collecting tiledbsoma
  Downloading tiledbsoma-1.11.1.tar.gz (291 kB)
...
Building wheels for collected packages: tiledbsoma
  Building wheel for tiledbsoma (pyproject.toml) ... |
...

-- it doesn't appear to be using the .whl file ...

Also note: on my arm64 mac with macos-14:

$ python -c 'from distutils import util; print(util.get_platform())'
macosx-14-arm64

However even macos12+arm64 runners in GHA misbehave:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9198915278/job/25302688619

distutil.util.get_platform: macosx-10.9-universal2

See also

• https://stackoverflow.com/questions/65888506/error-wheel-whl-is-not-a-supported-wheel-on-this-platform -- for the rename trick
• MacOS producing wheel for 10.9 even though 11.0 is specified pypa/cibuildwheel#952 -- nice to know, but this turns out not to help here
• Also note our CI YAML currently says pypa/[email protected] whereas newer versions exist https://github.com/pypa/cibuildwheel/releases/tag/v2.18.1 👀 🤔 ... this turns out not to help here